back down (v.)

in figurative sense of "withdraw a charge," 1859, American English, from notion of descending a ladder, etc. (the literal sense by 1849); from back (v.) + down (adv.).